Texture replacement in video sequences and images

ABSTRACT

Systems and methods for reducing bit rates by replacing original texture in a video sequence with synthesized texture. Reducing the bit rate of the video sequence begins by identifying and removing selected texture from frames in a video sequence. The removed texture is analyzed to generate texture parameters. New texture is synthesized using the texture parameters in combination with a set of constraints. Then, the newly synthesized texture is mapped back into the frames of the video sequence from which the original texture was removed. The resulting frames are then encoded. The bit rate of the video sequence with the synthesized texture is less than the bit rate of the video sequence with the original texture. Also, the ability of a decoder to decode the new video sequence is not compromised because no assumptions are made about the texture synthesis capabilities of the decoder.

The present application is a continuation of U.S. patent applicationSer. No. 11/179,701, filed on Jul. 12, 2005, which is a continuation ofU.S. patent application Ser. No. 10/237,489, filed on Sep. 9, 2002,which claims the benefit of U.S. Provisional Patent Application Ser. No.60/328,627, entitled “A Texture Replacement Method at the Encoder forBit Rate Reduction of Compressed Video,” filed Oct. 11, 2001, which isincorporated herein by reference.

RELATED APPLICATIONS

This application is related to commonly assigned U.S. patent applicationSer. No. 10/237,488, filed Sep. 9, 2002, now U.S. Pat. No. 7,606,435,issued Oct. 20, 2009, entitled “System And Method For Encoding AndDecoding Using Texture Replacement,” by Adriana Dumitras and BarinGeoffry Haskell and claiming priority to U.S. Provisional PatentApplication Ser. No. 60/360,027, filed Feb. 12, 2002. Thisabove-identified application is incorporated by reference herewith.

BACKGROUND OF THE INVENTION

1. The Field of the Invention

The present invention relates to systems and methods for reducing a bitrate of a video sequence. More particularly, the present inventionrelates to systems and methods for reducing a bit rate of a videosequence by replacing original texture of the video sequence withsynthesized texture at the encoder.

2. Background and Relevant Art

One of the goals of transmitting video sequences over computer networksis to have a relatively low bit rate while still maintaining a highquality video at the decoder. As technology improves and becomes moreaccessible, more users are leaving the realm of 56K modems and moving toDigital Subscriber Lines (DSL), including VDSL and ADSL, which support ahigher bit rate than 56K modems. VDSL, for example, supports bit ratesup to 28 Mbits/second, but the transmission distance is limited. Themaximum transmission distance for a 13 Mbits/second bit rate is 1.5 kmusing VDSL. ADSL, on the other hand, can support longer distances usingexisting loops while providing a bit rate of approximately 500kbits/second.

Video standards, such as MPEG-2, MPEG-4, and ITU H.263, can achieve bitrates of 3 to 9 Mbits/second, 64 kbits to 38.4 Mbits/second, and 8 kbitsto 1.5 Mbits/second, respectively. Even though video sequences with bitrates of hundreds of kbits/second can be achieved using these standards,the visual quality of these video sequences is unacceptably low,especially when the content of the video sequences is complex.

Solutions to this problem use model-based analysis-synthesis compressionmethods. Model-based analysis-synthesis compression methods perform bothanalysis and synthesis at the encoder to modify parameters in order tominimize the error between the synthesized model and the original. Theresulting parameters are transmitted to the decoder, which is requiredto synthesize the model again for the purpose of reconstructing thevideo sequence.

Much of the model-based analysis-synthesis compression methods havefocused on modeling human head-and-shoulders objects while fewerattempts have modeled background objects. Focusing on humanhead-and-shoulder objects often occurs because in many applications,such as videoconferencing applications, the background is very simple.However, background modeling may also achieve a significant reduction ofthe bit rate as the bit rate of I (intra) frames is often dependent onthe texture content of each picture. To a lesser extent, the bit rate ofB (bi-directionally predicted) frames and P (predicted) frames is alsoaffected by texture content as moving objects uncover additionalbackground objects.

One proposal for reducing the bit rate is to use sprite methods on thebackground objects. Sprites are panoramic pictures constructed using allof the background pixels that are visible over a set of video frames.Instead of coding each frame, the sprite is compressed and transmitted.The background image can be reconstructed using the sprite andassociated camera motion parameters. Sprite methods require exact objectsegmentation at the encoder, which is often a difficult task for complexvideo sequences. In addition, the motion or shape parameters that aretransmitted with the sprite consume some of the available bit rate.These limitations may be addressed by filtering the textured areas.Unfortunately, different filters must be designed for various textures.

Texture replacement has also been proposed as a method of backgroundmodeling. In one example, the original texture is replaced with anothertexture that is selected from a set of textures. However, this requiresthat the set of replacement textures be stored at the encoder. Inanother example, the texture of selected regions is replaced at theencoder with pixel values that represent an “illegal” color in the YUVcolor space. At the decoder, the processed regions are recovered usingchroma keying. There is an explicit assumption that texture synthesis,using texture parameters sent from the encoder, followed by mapping ofthe synthesized texture onto the decoded video sequences, is performedat the decoder. This method therefore assumes that the reconstruction isperformed at the decoder using a method that is dependent on thedecoder's processing capabilities. The drawbacks of these approaches arethat the processing capabilities of the decoder are assumed and that thecomputational costs of the decoding stage are increased.

BRIEF SUMMARY OF THE INVENTION

These and other limitations of the prior art are overcome by the presentinvention which relates to systems and methods for reducing the bit rateof a video sequence through texture replacement at the encoder. Thecapabilities of the decoder are not assumed and the decoder is notrequired to perform texture analysis and synthesis. As would be known inthe art, an encoder and a decoder are computing devices each having aprocessor that is controlled by software instructions or softwaremodules. The synthesized texture that replaces the original texture hassimilar perceptual characteristics to the original texture. Thus, thevideo sequence with the synthesized texture is visually similar to theoriginal video sequence. The bit rate of the video sequence withsynthesized textures is reduced because the synthesized textures thathave replaced the original textures can be coded more effectively.

Texture replacement, in accordance with the present invention, occurs atthe encoder and is therefore independent of the capabilities of thedecoder. Texture replacement begins by selecting and removing texturefrom some or all of the original frames in the video sequence. Theremoved texture is analyzed to obtain texture parameters. Then, newtexture is synthesized using the texture parameters in combination witha set of qualitative constraints. The synthesized texture can becompressed more effectively than the original texture and is alsosimilar to, yet distinguishable from, the original texture. After thenew texture is synthesized, the synthesized texture is inserted backinto the original frames and the video sequence that includes thesynthesized texture is encoded. Advantageously, the bit rate of thecompressed video sequence with the synthesized texture is lower than thebit rate of the compressed video sequence with the original texture.

Additional features and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by the practice of the invention. Thefeatures and advantages of the invention may be realized and obtained bymeans of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present inventionwill become more fully apparent from the following description andappended claims, or may be learned by the practice of the invention asset forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the invention can be obtained, a moreparticular description of the invention briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only typical embodiments of the invention and are not thereforeto be considered to be limiting of its scope, the invention will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 is a flow diagram illustrating an exemplary method for reducing abit rate of a video sequence by replacing original texture withsynthesized texture;

FIG. 2 illustrates a frame that includes various candidate textureswhich can be replaced with synthesized textures;

FIG. 3 is a flow diagram of a method for removing original texture fromvarious frames of a video sequence;

FIG. 4 illustrates a recursive transform used in image decomposition toobtain texture parameters; and

FIG. 5 illustrates a basic system or computing device embodiment of theinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention relates to systems and methods for texturereplacement in video sequences and to reducing bit rates of videosequences through texture replacement. The bit rate of a compressedvideo sequence is reduced because the synthesized texture can be moreeffectively compressed than the original texture. Because the texturereplacement occurs as part of the encoding process, the computationalcosts of the decoder are not increased and the processing/synthesizingcapabilities of the decoder are not assumed.

FIG. 1 is a flow diagram that illustrates one embodiment of a method forreplacing texture in a video sequence and for reducing the bit rate ofthe encoded video sequence. Reducing the bit rate of a video sequencebegins by removing selected texture from a frame or from a series offrames included in the video sequence (102). Typically, texture from asingle frame is identified, while the identified texture from thatparticular frame is removed from a set of frames. This eliminates theneed to identify texture in each frame, which would increase thecomputational costs of texture replacement. After the original texturehas been removed from the selected frame, the removed texture isanalyzed (104) to obtain texture parameters. The texture parameters, incombination with a set of constraints, are used to generate synthesizedtexture (106). The synthesized texture is then inserted into theoriginal frames (108) from which the original texture was removed.

One advantage of texture replacement is that the synthesized texture canbe compressed more effectively than the original texture. The ability tomore effectively compress the synthesized texture results in a reducedbit rate for the encoded video sequence. The constraints are appliedduring the synthesis of the new texture to ensure that the synthesizedtexture is similar to the original texture. This is useful, for example,in order to retain the artistic representation of the original frames.However, the present invention does not require that the synthesizedtexture resemble the original texture.

FIGS. 2 and 3 more fully illustrate how texture is selected and removedfrom the original frames of a video sequence. FIG. 2 illustrates anexemplary frame 200. The frame 200 is a picture that, in this example,includes a person 202, a person 203 and a background. The frame 200includes several textures that can be selected for removal. Texturerefers to those portions or regions of a frame that have the same orsimilar characteristics and/or appearance. Usually, the background isthe primary source of texture, although the present invention is notlimited to background textures. One reason for selecting backgroundtextures is that these same textures or regions are present in multipleframes and in substantially the same locations. For example, if nomovement is present in a particular set of frames, the background willremain constant across the set of frames. However, the systems andmethods described herein can be applied even when a set of framesincludes moving objects.

In this example, the background includes several candidate textures thatcan be selected for removal. The background of the frame 200 includesthe ground texture 204, the sky texture 206 and the cloud texture 208.Each of these objects in the background are examples of textures. It islikely that the region of the frame 200 covered by the ground texture204, the sky texture 206, or the cloud texture 208, respectively, havesimilar appearances or characteristics.

When a region of texture is selected for removal, it is useful to selecta texture that has a spatial region of support that covers a reasonablepart of the frame area over a sufficiently large number of frames. Inone embodiment, a texture that covers an area that is larger than 30% ofthe frame area is typically selected, although the present invention canapply to any texture, regardless of the frame area occupied by theselected texture. In addition, selecting a texture that can be replacedin a large number of frames is advantageous because it has an impact onthe bit rate of the encoded video sequence.

It is also useful to select a texture that belongs to a class oftextures that are amenable to replacement. Generally, candidate texturesinclude regions of a frame that have similar characteristics and/orappearance. Exemplary textures include natural textures (foliage, grass,ground, water, sky, building facades, etc.), and the like that can beidentified in video sequences or test sequences such as movie sequences.Frames with replaceable texture typically have absent or slow globalmotion and/or only a few moving objects. Thus, the present invention canbe applied to particular frame sequences within a video sequence anddoes not have to be applied to the entire video sequence.

Texture Removal and Region Segmentation

FIG. 3 illustrates a flow diagram illustrating how a texture is selectedand removed from the frame illustrated in FIG. 2. First, a region ofinterest (ROI) 210 is selected within the texture to be replaced (302).In this example, the selected ROI 210 is within the ground texture 204.The ROI 210 typically includes more than one pixel. The ROI 210, forexample, may include a 7×7 array of pixels. After the ROI 210 isselected, the color characteristics of the ROI 210 are compared to thecolor characteristics (304) of all pixels within the frame 200. Thepixels that have similar characteristics are classified or identified asbelonging to the same region (306) as the ROI 210. The pixels identifiedin this manner are thus included in the texture that will be removedfrom the frame(s). The pixels that are in the identified region areremoved and are temporarily replaced with pixels that have an arbitraryvalue. In one embodiment, the replacement pixels have a constant valueequal to the mean color of the selected and identified region.

More particularly in one embodiment, region segmentation and textureremoval occurs in stages. For example, let a color frame be representedby a set of two-dimensional planes in the YUV color space. Each of theseimage planes is represented as a matrix and each matrix element includesa pixel value in row i and column j. More specifically, the Y framesconsist of all of the pixels {(i, j), with 1≦i≦M, 1≦j≦N}. First,original frames from the YUV color space are converted to the RGB colorspace. Second, the location {i_(r), j_(r)} is selected from aregion-of-interest, such as the ground texture 204 shown in FIG. 2, andthe ROI is constructed from:

$\begin{matrix}{{{ROI}_{r} = \left\{ {{i = {i_{r} + k_{i}}},{j = {j_{r} + k_{j}}}} \right\}},{{{with} - \left\lbrack \frac{w_{r}}{2} \right\rbrack} \leq k_{i}},{k_{j} \leq \left\lbrack \frac{w_{r}}{2} \right\rbrack},} & (1)\end{matrix}$where the operator [ ] denotes “the integer part of” and w_(r) is odd.Alternatively, the region-of-interest of a size equal to w_(r)×w_(r)pixels can be manually selected from a region of interest. The pixelvalues are smoothed by applying an averaging filter to the ROI 210 ineach of the R, G and B color planes, and the mean vector [μ_(r)^(R)μ_(r) ^(G)μ_(r) ^(B)] is computed, where μ_(r) ^(R), μ_(r) ^(G), andμ_(r) ^(B) stand for the mean values within the ROI in the R, G, and Bcolor planes, respectively.

Next, the angular map and the modulus map of the frame are computed asfollows:

$\begin{matrix}{{{\theta\left( {i,j} \right)} = {1 - {\frac{2}{\pi}{arc}\;{\cos\left( \frac{v_{({i,j})}v_{ref}}{{v_{({i,j})}}{v_{ref}}} \right)}}}},{and}} & (2) \\{{{\eta\left( {i,j} \right)} = {1 - \frac{{v_{({i,j})} - v_{ref}}}{\sqrt{3 \times 255^{2}}}}},} & (3)\end{matrix}$respectively, with respect to a reference color vector. The followingworks more fully describe angular and modulus maps and are herebyincorporated by reference: (1) Dimitris Androutsos, Kostas Plataniotis,and Anastasios N. Venetsanopoulos, “A novel vector-based approach tocolor image retrieval using a vector angular-based distance measure,”Computer Vision and Image Understanding, vol. 75, no. ½, pp. 46-58,July/August 1999; (2) Adriana Dumitras and Anastasios N.Venetsanopoulos, “Angular map-driven snakes with application to objectshape description in color images,” accepted for publication in IEEETransactions on Image Processing., 2001; and (3) Adriana Dumitras andAnastasios N. Venetsanopoulos, “Color image-based angular map-drivensnakes,” in Proceedings of IEEE International Conference on ImageProcessing, Thessaloniki, Greece, October, 2001. Notations v(_(i,j)) andv_(ref) stand for a color vector [R(i,j) G(i,j) B(i.j)] in the RGB colorspace, and the reference vector that is selected to be equal to [μ_(r)^(R)μ_(r) ^(G)μ_(r) ^(B)], respectively. The notation θ stands for thevalue of the angle given by (2) between the vector v(_(i,j)) and thereference vector v_(ref). The notation η stands for the value of themodulus difference given by (3) between the vector v(_(i,j)) and thereference vector v_(ref), respectively.

In order to identify the pixels that have similar color characteristicsto those of the reference vector, the distance measure is computed byd ^(θη)(i,j)=exp [−θ(i,j)η(i,j)]  (4)and the mean distance given is computed by

$\begin{matrix}{{{\mu\frac{d}{r}} = {ɛ\left\{ {d^{\theta\eta}\left( {i,j} \right)} \right\}}},} & (5)\end{matrix}$where notation ε stands for a mean operator over ROI_(r). All of thepixels within the frame that satisfy the constraint

$\begin{matrix}{\left( {{d^{\theta\;\eta}\left( {i,j} \right)} - {\mu\frac{d}{r}}} \right)^{2} < {ɛ\; c}} & (6)\end{matrix}$are clustered into regions.

Next, the segmented regions that satisfy the constraint of (6) areidentified and labeled. In one embodiment, all regions with a normalizedarea A_(R) smaller than a threshold area, i.e.,

$\frac{Ar}{MN} \leq {ɛ\; A}$where M×N is the frame area, are discarded. The remaining regions arelabeled. If a segmentation map consisting of all of the segmentedregions is not considered acceptable by the user, another ROI locationis selected and processed as described above. The labeled regions areremoved from the frame and texture removal of the current frame iscomplete. The frames having the texture and color removed using thesegmentation map are obtained at the end of the texture removal stage.

To summarize, the region segmentation and texture removal proceduresdescribed above identify pixels in the original frame that have similarcharacteristics in terms of color to those of the pixel (location) orROI selected by the user. The color characteristics of the identifiedpixels or segmented regions are evaluated using an angular map and amodulus map of the color vectors in the RGB color space. Angular mapsidentify significant color changes within a picture or frame, whichtypically correspond to object boundaries, using snake models. Toidentify color changes or boundaries, simple gray-level edge detectionis applied to the angular maps. Identification of major change ofdirection in vector data and color variation computation by differencesbetween inside and outside contour points can also be applied. In thepresent invention, however, a classification of the pixels is performedin each frame using the angular map. Despite the fact that theperformance of the color-based region segmentation stage depends on theselection of the threshold values εC,εA and the size of the ROI, thevalues of these parameters may be maintained constant for various videosequences.

Texture Analysis

Analyzing the removed texture includes computing a set of textureparameters from the removed texture. In one embodiment, a parametricstatistical model is employed. This model, which employs an overcompletemultiscale wavelet representation, makes use of steerable pyramids forimage decomposition. Steerable pyramids as known in the art are morefully discussed in the following article, which is hereby incorporatedby reference: Javier Portilla and Eero P. Simoncelli, “A parametrictexture model based on joint statistics of complex waveletcoefficients,” International Journal of Computer Vision, vol. 40, no. 1,pp. 49-71, 2000. The statistical texture descriptors are based on pairsof wavelet coefficients at adjacent spatial locations, orientations andscales (in particular, the expected product of the raw coefficient pairsand the expected product of their magnitudes), pairs of coefficients atadjacent scales (the expected product of the fine scale coefficient atadjacent scale coefficient), marginal statistics and lowpasscoefficients at different scales.

First, a steerable pyramid decomposition is obtained by recursivelydecomposing the texture image into a set of oriented subbands andlowpass residual band. The block diagram of the transform is illustratedin FIG. 4, where the area enclosed by the dashed box 400 is insertedrecursively at the point 402. Initially, the input image is decomposedinto highpass and lowpass bands using the exemplary filters

${L_{0}\left( {r,\theta} \right)} = \frac{L\left( {\frac{r}{2},\theta} \right)}{2}$and${H_{0}\left( {r,\theta} \right)} = {{H\left( {\frac{r}{2},\theta} \right)}.}$The lowpass band is then decomposed into a lower frequency band and aset of oriented bands. The filters used in this transformation arepolar-separable in the Fourier domain and are given by:

${L\left( {r,\theta} \right)} = \left\{ {\begin{matrix}{{2{\cos\left( {\frac{\pi}{2}{\log_{2}\left( \frac{4r}{\pi} \right)}} \right)}},} & {\frac{\pi}{4} < r < \frac{\pi}{2}} \\{2,} & {r \leq \frac{\pi}{4}} \\{0,} & {r \geq \frac{\pi}{2}}\end{matrix},{{{and}{B_{k}\left( {r,\theta} \right)}} = {{H(r)}{G_{k}(\theta)}}},{{k\; ɛ\left\lfloor {{0_{\xi}K} - 1} \right\rfloor{where}{H(r)}} = \left\{ {\begin{matrix}{{\cos\left( {\frac{\pi}{2}{\log_{2}\left( \frac{2r}{\pi} \right)}} \right)},} & {\frac{\pi}{4} < r < \frac{\pi}{2}} \\{1,} & {r \geq \frac{\pi}{2}} \\{0,} & {r \leq \frac{\pi}{4}}\end{matrix},{{{and}{G_{k}(\theta)}} = \left\{ \begin{matrix}{{2^{\kappa - 1}{\frac{\left( {K - 1} \right)!}{\sqrt{{K\left\lbrack {2\left( {K - 1} \right)} \right\rbrack}!}}\left\lbrack {\cos\left( {\theta - \frac{\pi\; K}{K}} \right)} \right\rbrack}^{K - 1}},} & {{{\theta - \frac{\pi\;\kappa}{K}}} < \frac{\pi}{2}} \\{0,} & {{otherwise}.}\end{matrix} \right.}} \right.}} \right.$

The notations r and θ stand for polar coordinates in the frequencydomain, K denotes the total number of orientation bands.

Statistical texture descriptors are computed using the imagedecomposition previously obtained. More specifically, marginalstatistics, correlations of the coefficients, correlations of thecoefficients' magnitudes and cross-scale phase's statistics arecomputed. In terms of marginal statistics (a) the skewness and kurtosisof the partially reconstructed lowpass images at each scale, (b) thevariance of the highpass band, (c) the mean variance, skewness andkurtosis and (d) the minimum and maximum values of the image pixels(range) are computed at each level of the pyramid. In terms ofcoefficient correlations, the autocorrelation of the lowpass imagescomputed at each level of the pyramid decomposition is computed. Interms of magnitude correlation, the correlation of the complex magnitudeof pairs of coefficients at adjacent positions, orientations and scalesis computed. More specifically, (e) the autocorrelation of magnitude ofeach subband, (f) the crosscorrelation of each subband magnitudes withthose of other orientations at the same scale, and (g) thecrosscorrelation of subband magnitudes with all orientations at acoarser scale are obtained. Finally, in terms of cross-scale statistics,the complex phase of the coarse-scale coefficients is doubled at allorientations and then the crosscorrelation between these coefficientsand the fine-scale coefficients is computed. Doubling of thecoarse-scale coefficients is motivated by the fact that the local phaseof the responses to local feature such as edges or lines changes at arate that is, for fine-scale coefficients, twice the rate of that of thecoefficients at a coarser scale.

Texture Synthesis

During texture synthesis, a synthesized texture is created that can becompressed more effectively than the original texture. In oneembodiment, the synthesized texture is similar to, yet distinguishablefrom, the original texture. Because the synthesized texture can becompressed more effectively, the bit rate of the compressed frames withsynthesized texture is lower than the bit rate of the compressed frameswith the original texture. Retaining the visual similarity ensures thatthe frames with synthesized texture convey the same artistic message asthat of the frames with the original texture.

A set of qualitative texture synthesis constraints are used to achievevisual similarity between the synthesized texture and the originaltexture. The texture parameters selected for texture synthesis arederived using the set of qualitative constraints.

In one embodiment, the synthesis of a new texture that is visuallysimilar to the original texture is subject to constraints in terms ofdominant texture orientation, and overall color and color saturation.Exemplary constraints include, but are not limited to:

(C1) Marginal statistics;

(C2) Coefficient correlations;

(C3) Coefficient magnitude correlations;

(C4) Cross-scale statistics;

(C5) Overall color; and

(C6) Color saturation.

If the original texture that has been removed from the original framesis structured and busy, the synthesized texture is subject to theconstraints C1, C2, C3, C4, C5 and C6. If the original texture isunstructured and smooth, the synthesized texture is subject to theconstraints C3, C4, C5 and C6.

Using these constraints, the new texture is synthesized by firstdecomposing an image containing Gaussian white noise using a complexsteerable pyramid. Next, a recursive coarse-to-fine procedure imposesthe statistical constraints on the lowpass and bandpass bands whilesimultaneously reconstructing a lowpass image. In one embodiment of thepresent invention, the order in which the constraints are applied is C3,C4, C2, C1, C5, and C6 for structured and busy textures, and C3, C4, C5,and C6 for unstructured and smooth textures.

The texture synthesis constraints are derived from the basic categories(vocabulary) and rules (grammar) used by humans when judging thesimilarity of color patterns. The following work, which more fullydescribes the vocabulary and grammar of color patterns is herebyincorporated by reference: Aleksandra Mojsilovic, Jelena Kovacevic,Jianying Hu, Robert J. Safranek, and S. Kicha Ganapathy, “Matching andretrieval based on the vocabulary and grammar of color patterns,” IEEETransactions on Image Processing, vol. 9, no. 1, pp. 38-54, January2000. In one embodiment, the basic pattern categories used by humans forsimilarity evaluation of color patterns are directionality, regularityand placement, complexity and heaviness and the basic color categoriesused by humans for the same task are overall color and color purity. Thefollowing table provides the meanings for these criteria:

Derived Criterion for Similarity constrained texture Criterion Expressessynthesis Overall color Perception of a single dominant color, or aPreserve the overall multicolored image that creates the impressioncolor in the synthesized of a dominant color texture Directionality andPerception of the dominant orientation in edge Preserve the dominantorientation distribution, or the dominant direction in the orientationin the repetition of the structural element synthesized textureRegularity and perception of the placement (and small Not appliedplacement perturbation in placement), repetition and uniformity of thepatterns Color purity Perception of the degree of colorfulness inPreserve the color (overall patterns saturation in the saturation)synthesized texture Pattern Perception of a general impression based onPreserve the type of the complexity and the type of overall color (lightversus dark) or overall color (light or heaviness the overall chroma(saturated versus dark) unsaturated) or the spatial frequency in therepetition of the structural element or the color contrast

Because different visual pathways process patterns and colors in thehuman visual system, separate constraints for texture and color arederived in the texture synthesis process. Moreover, texture is typicallysynthesized with gray levels, i.e. luminance frames, and colorconstraints are imposed on the chrominance frames. In terms of texture,the synthesized texture should have similar dominant directionality asthat of the original texture. In terms of color, the synthesized textureshould have similar overall color and color saturation as those of theoriginal texture.

To address the texture requirement, the magnitude correlation constraint(C3) which represents the structure in images and the cross-scalestatistics constraint (C4) which allows distinguishing lines and edgesare selected and applied. While these texture constraints are sufficientfor unstructured and smooth textures, they do not allow synthesizingstructured and busy textures with appearances that are similar to thoseof the original ones. Therefore, for structured and busy textures, themarginal statistics constraint (C1) and the coefficient correlationconstraint (C2) are used. The correlation constraint characterizes theregularity of the texture as represented by periodic or globallyoriented structure in the set of constraints. The overall color (C5) andcolor saturation (C6) are the color constraints that address the colorrequirements. The overall color constraint may be further expressed interms of similar color moments (mean color and standard colordeviation). Because the pixel values in the segmented regions have beenreplaced with mean pixel values within the region, the overall color iseasily preserved. Because preserving only the mean color would yieldtextures with discolored appearances in the cases of textures thatexhibit large color variations, the color situations are also preserved.

By selecting different sets of constraints for unstructured and smoothtextures, and structured and busy textures, different characteristics ofthe background textures that are present in video sequences can beutilized. For example, by removing the marginal statistics constraint(C1), synthesized textures that differ in their first and second orderstatistics from the original textures can be obtained. Consequently,according to Julesz's conjecture, the synthesized and original textureswould then be distinguishable in pre-attentive (undetailed) evaluationof less than 50 milliseconds (ms). At a frame rate of 24 frames persecond, each frame is indeed viewed in a pre-attentive mode, i.e., forapproximately 40 (<50) ms. By removing the coefficient correlationconstraint (C2) for the same class of unstructured and smooth textures,the requirement that the synthesized and the original textures havesimilar periodicity is relaxed. This, in turn, improves the compressioneffectiveness.

The texture synthesized is mapped on the luminance frame of the videosequence. The segmented regions within the chrominance frames remainfilled with pixels having mean values as obtained by the regionsegmentation. Thus, the color constraints are satisfied.

Although the present invention has been described in terms of a videosequence, it is understood by one of skill in the art that the systemsand methods described herein also apply to a still image or to a singleframe. Thus removing texture from a set of frames includes the casewhere the set of frames is a single frame or a still image.

One example of a system or apparatus which illustrates a hardwareimplementation of the second embodiment of the invention is shown inFIG. 5. With reference to FIG. 5, an exemplary system for implementingthis embodiment of the invention includes a general-purpose computingdevice 500, including a processing unit (CPU) 520 and a system bus 510that couples various system components including the system memory suchas read only memory (ROM) 540 and random access memory (RAM) 550 to theprocessing unit 520. Other system memory 530 may be available for use aswell. A basic input/output (BIOS), containing the basic routine thathelps to transfer information between elements within the computingdevice 500, such as during start-up, is typically stored in ROM 540. Thecomputing device 500 further includes storage means such as a hard diskdrive 560, a magnetic disk drive, an optical disk drive, tape drive orthe like. The storage device 560 is connected to the system bus 510 by adrive interface. The drives and the associated computer readable mediaprovide nonvolatile storage of computer readable instructions, datastructures, program modules and other data for the computing device 500.The basic components are known to those of skill in the art andappropriate variations are contemplated depending on the type of device,such as whether the device is a small, handheld computing device, adesktop computer, or a computer server.

To enable user interaction with the computing device 500, an inputdevice 590 represents an input mechanism for user input. The deviceoutput 570 can also be one or more of a number of output means. Thecommunications interface 580 generally governs and manages the userinput and system output.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. A method comprising: removing a region ofinterest belonging to a background portion of an initial frame in avideo sequence; determining, via a processor, texture parametersassociated with the region of interest; synthesizing replacement textureusing the texture parameters; modifying the replacement texture byapplying a set of constraints to the replacement texture, wherein theset of constraints comprises a first set of constraints when the regionof interest is structured and a second set of constraints when theregion of interest in unstructured, the first set of constraints havinga sequence of application which is distinct from the second set ofconstraints, wherein the first set of constraints shares a dominanttexture orientation constraint with the second set of constraints, toyield new texture; and inserting the new texture into the videosequence.
 2. The method of claim 1, further comprising: encoding thevideo sequence containing the new texture, to yield an encoded videosequence.
 3. The method of claim 2, wherein a modified bit rateassociated with the encoded video sequence is lower than an originaltexture bit rate associated with the video sequence.
 4. The method ofclaim 1, wherein the new texture is similar to, but distinguishablefrom, the region of interest.
 5. The method of claim 1, furthercomprising: selecting the region of interest based on a percentage ofarea within the initial frame; and wherein the first set of constraintsfurther shares a color saturation constraint with the second set ofconstraints.
 6. The method of claim 1, wherein the second set has anumber of constraints less than a number of constraints in the firstset.
 7. The method of claim 1, wherein the texture parameters aredetermined using a parametric statistical model.
 8. A system comprising:a processor; and a computer readable storage medium having instructionsstored which, when executed by the processor, cause the processor toperform operations comprising: removing a region of interest belongingto a background portion of an initial frame in a video sequence;determining, via a processor, texture parameters associated with theregion of interest; synthesizing replacement texture using the textureparameters; modifying the replacement texture by applying a set ofconstraints to the replacement texture, wherein the set of constraintscomprises a first set of constraints when the region of interest isstructured and a second set of constraints when the region of interestin unstructured, the first set of constraints having a sequence ofapplication which is distinct from the second set of constraints,wherein the first set of constraints shares a dominant textureorientation constraint with the second set of constraints, to yield newtexture; and inserting the new texture into the video sequence.
 9. Thesystem of claim 8, the computer readable storage medium havingadditional instructions stored which result in the operations furthercomprising: encoding the video sequence containing the new texture, toyield an encoded video sequence.
 10. The system of claim 9, wherein amodified bit rate associated with the encoded video sequence is lowerthan an original texture bit rate associated with the video sequence.11. The system of claim 8, wherein the new texture is similar to, butdistinguishable from, the region of interest.
 12. The system of claim 8,wherein the computer readable storage medium having additionalinstructions stored which result in the operations further comprising:selecting the region of interest based on a percentage of area withinthe initial frame; and wherein the first set of constraints furthershares a color saturation constraint with the second set of constraints.13. The system of claim 8, wherein the second set has a number ofconstraints less than a number of constraints in the first set.
 14. Thesystem of claim 8, wherein the texture parameters are determined using aparametric statistical model.
 15. A computer-readable storage devicehaving instructions stored which, when executed on a computing device,cause the computing device to perform operations comprising: removing aregion of interest belonging to a background portion of an initial framein a video sequence; determining, via a processor, texture parametersassociated with the region of interest; synthesizing replacement textureusing the texture parameters; modifying the replacement texture byapplying a set of constraints to the replacement texture, wherein theset of constraints comprises a first set of constraints when the regionof interest is structured and a second set of constraints when theregion of interest in unstructured, the first set of constraints havinga sequence of application which is distinct from the second set ofconstraints, wherein the first set of constraints shares a dominanttexture orientation constraint with the second set of constraints, toyield new texture; and inserting the new texture into the videosequence.
 16. The computer-readable storage device of claim 15 storingadditional instructions which result in the method further comprising:encoding the video sequence containing the new texture, to yield anencoded video sequence.
 17. The computer-readable storage device ofclaim 16, wherein a modified bit rate associated with the encoded videosequence is lower than an original texture bit rate associated with thevideo sequence.
 18. The computer-readable storage device of claim 15,wherein the new texture is similar to, but distinguishable from, theregion of interest.
 19. The computer-readable storage device of claim15, the computer-readable storage device having additional instructionsstored which result in the operations further comprising: selecting theregion of interest based on a percentage of area within the initialframe; and wherein the first set of constraints further shares a colorsaturation constraint with the second set of three constraints.
 20. Thecomputer-readable storage device of claim 15, wherein the second set hasa number of constraints less than a number of constraints in the firstset.